DOCUMENT RESUME 

ED 383 299 IR 017 154 



AUTHOR 
TITLE 
PUB DATE 
NOTE 



PUB TYPE 



Harvey, Francis A.; Nelson, Adam 

A Hybrid Investigation of Hypermedia Training. 

95 

lip.; In: Proceedings of the 1995 Annual National 
Convention of the Association for Educational 
Communications and Technology (AECT) , (17th, Anaheim, 
CA, 1995) ; see IR 017 139. 
Reports - Evaluative/Feasibility (142) — 
Speeches/Conference Papers (150) 



EDRS PRICE 
DESCRIPTORS 



IDENTIFIERS 



MF01/PC01 Plus Postage. 

Adult Education; Data Analysis; Dropout Programs; 
Elementary Secondary Education; '''Hypermedia ; 
Instructional Design; '''Instructional Effectiveness; 
'''Management Development; '''Qual i tat i ve Research; 
Trainees; Trainers; '''Training Methods; '''Use 
Studies 

'■Cities in Schools 



ABSTRACT 

This document describes research which investigated 
the effectiveness of the Cities in School (CIS) hypermedia training 
course for senior executives of that national dropout prevention 
program. The study employed a hybrid methodology of qualitative 
techniques and computer tracking tools, in order to provide a 
complete picture of how the hypermedia training course was delivered 
and received. Thirteen trainees and three professional trainers 
participated in a week-long training course. The goals of the study 
included how hypermedia is used in a "real-world" setting; hypermedia 
characteristics for further research; how frequently and to what 
extent trainers and trainees used hypermedia; methods of exploring 
hypermedia materials; and relationships of hypermedia use, 
non-hypermedia activities, and attainment of learning objectives of 
the course. Results of the qualitative component indicate that the 
following areas warrant examination in future studies: diversity of 
trainers and grouping issues; cooperative learning; presentation, 
formal evaluation of learning; applying learning to real-world 
situations; use of supplemental trainer-developed materials and print 
materials; variations in trainers' styles; time management; and 
course evaluation. The preliminary results of the computer tracking 
component of the study are presented, but it was not possible to 
continue the analysis of the relationship between uses of hypermedia 
and the attainment of specific learning objectives. Preliminary 
analysis of hypermedia paths used by trainers indicates that the 
trainees often traveled extremely complex paths while using the 
1 ypermedia training materials. Implementing computer tracking tools 
requires considerable programming expertise and effort, and are most 
efficiently implemented if they are incorporated into the design of 
the hypermedia application as it is developed. (Contains 8 
references.) (AEF) 
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Introduction 



Hypermedia is a relatively new technology, and data on the most effective ways to use hypermedia 
in training, while steadily growing, are still relatively scattered. Hypermedia, because of its greatly 
expanded capabilities over traditional computer-based training, presents new challenges for designers, 
developers, and evaluators of training as well as trainers. Issues arise in the design, development, and 
delivery of hypermedia training which were not present with traditional computer-based training and 
therefore were not given extensive treatment either in the development of theoretical models of computer- 
based instructional design or in the design of such instructional materials. Examples of the additional 
complexities which must be addressed in designing and delivering effective hypermedia-based training 
include: the appropriate type and degree of structure to be incorporated into the information content of 
hypermedia applications; the extent and type of navigation options to be provided to the user, the 
appropriate role of the trainer or facilitator; and the appropriate mix of different media (text, computer 
graphics, computer-generated animation, still video graphics and photographs, motion video, digitized 
speech, other audio, etc.). 

Use of Qualitative Research Methodologies 

Because hypeimedia training is so new and so complex, it is difficult to identify appropriate 
research hypotheses which can be investigated using quasi-experimental techniques. Furthermore, the 
complexity of hypermedia training used in trainer-facilitated group settings (the type of hypermedia training 
investigated in this study) makes it difficult if not impossible to control adequately for plausible rival 
hypotheses. For these reasons, these researchers feel that qualitative methodologies constitute a more 
appropriate approach for the investigation of the effectiveness of hypermedia training. 

Qualitative approaches provide a broader perspective on actual applications of hypermedia training 
in real-world settings. This approach was chosen because the researchers recognized that, at this stage in the 
development of hypermedia technology, we do not have a sufficiently well-developed understanding of 
which variables related to the use of hypermedia are most appropriate to investigate, and how to 
systematically control these variables in quasi-experimental studies. Use of qualitative methodologies also 
represents an acknowledgment that the researcher can not, indeed may not want to, exert complete control 
over the training situation. In fact, attempts to impose undue control over the delivery of training may 
introduce distortions in the way hypermedia is used which will negatively effect the validity of the data 
collected. At this point in the development of our understanding of how to use hypermedia effectively for 
training, it seems that the most important question to be addressed is, "What is going on when hypermedia 
is used for training?" 

Use of Computer Tracking Tools 

On the other hand, quantitative data on the uses of hypermedia for training, when they can be 
collected unobtrusively and without artificially distorting the delivery of training, can provide the foundation 
for analysis which supplements the qualitative observations iuid contributes to their validity, or which flags 
results which may appear to be conflicting and which require further investigation. Of particular value is 
the collection of quantitative data on the extent and specific ways in which trainees use hypermedia in a 
specific training program. This can be accomplished through the use of computer-based tracking tools 
incorporated into the hypermedia application used. 

Computer tracking tools can record the number and type of inputs (keyboard entry, mouse pointing 
or clicking, etc.) into the hypermedia program by the user at any point in the program, as well as the exact 
time and duration of the interaction. This information, then, provides a comprehensive record of how a 
specific user interacted with the hypermedia, including the paths taken by the user from screen to screen, and 
the type, time, and duration of the interactions. 

Comp uter tracking tools have been used by a number of researchers for formative evaluation and 
for formal research on attributes of hypermedia training. Gay and Mazur (1993) described several different 
types of computer tracking tools, ranging from tools which produce primarily text-based lists to tools 
which produce graphical and/or pictorial depictions of users' actions. They described how they used these 
tools in the investigation of hypermedia programs developed at Cornell's Interactive Multimedia Group. 
Misanchuk and Schwier (1991) identified three distinct uses for computer tracking tools, the outcomes of 
which they referred to as audit trails. These uses included: 1) date collection for formative evaluation; 2) 
tools for basic research into instructional design of multimedia; 3) auditing use of multimedia in a public 
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fonim. They describe several quantitative and qualitative methods for describing and analyzing computer- 
generated audit trails. 

The developers of the Cities in Schools (CIS) hypermedia training curriculum recognized the 
potential of incorporating computer tracking tools into the CIS hypermedia software early in the project. 
Design specifications of the hypermedia included an analysis of the types of data to be collected and the 
mechanisms for collecting that data automatically (Story and Harvey, 1990). 

Use of Hybrid Methodologies 

This paper will describe a research effort which investigated the effectiveness of the CIS 
hypermedia training course for senior executives of that national dropout prevention program. The study 
employed a hybrid methodology of qualitative techniques and computer tracking tools, in order to provide as 
complete a picture as possible of how the hypermedia training course was delivered, and received. 



The Cities in Schools Hypermedia Training Curriculum 

The authors have been involved, for the past six years, in the design, development, and 
implementation of an integrated hypermedia training curriculum for Cities in Schools, Inc. The CIS 
training curriculum v.ij developed by faculty and staff of the Advanced Information Technologies 
Laboratory (AITL) in Lehigh University's College of Education. 

The CIS integrated hypermedia training curriculum was developed by a team which included 
instructional design specialists; designers and producers of video, computer, and print training materials; and 
experts in adult learning. Three courses, comprising a total of ten weeks of full-day training, have been 
developed. The CIS training curriculum is now being delivered by CIS trainers (trained by AITL staff) at 
the CIS National Training Institute at Lehigh University and at six regional CIS training centers around the 
country. 

The Cities in Schools hypermedia training curriculum uses a training paradigm different from the 
one-person/one computer paradigm of traditional computer-based training. Instead, a novel approach to the 
use of hypermedia for training is employed in which the hypermedia materials are used as training resources 
to facilitate interpersonal interactions in trainer-facilitated group learning activities. 

The CIS training course investigated for this study was the Program Operations course, designed to 
train senior management personnel who are executive directors of city-wide or county-wide CIS programs. 
The course included a comprehensive (2500-screen) hypermedia software program, a one-hour videodisk, and 
trainers' and trainees' printed materials. A simulated city with a CIS dropout prevention program were 
incorporated into the training materials. A trainer's resource book included objectives, learning strategies, 
and evaluation strategies for 32 learning activities which incorporated a total of 1 17 separate learning sub- 
activities. 

The CIS Program Operations course supported multiple modes of learning, including: 

• presentations by the trainer; 

• examination of computer, print, and/or video materials by individ .s'.s, pairs, or small groups of 
trainees; 

• small group and large group discussions; 

• role playing; 

• other group activities, and 

• individualized, self-paced learning activities. 

Design, development and delivery of CIS hypermedia training was guided by the Advanced 
Information Technologies Laboratory Process Model (Harvey, 1993). According to this model, learning 
proceeds from lower level activities (information acquisition and comprehension) to higher level (analysis 
and synthesis; application in simulated settings), and culminates in a systematic application of the 
knowledge to a real-world situation. 



Purpose of the Study 

The authors recognized that, since hypermedia is a new tool for training, a comprehensive body of 
knowledge describing how hypermedia can best be used for training has not yet been developed. In fact, the 
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salient characteristics of an appropriate training program which utilizes hypermedia training have not yet 
been identified. The Cities in Schools hypermedia training course represented the collective knowledge and 
wisdom of the AITL development group. The purpose of the research study reported in this paper was fo 
validate the approach taken in developing CIS hypermedia training, and to identify aspects of using 
hypermedia for training which would warra' ser examination. Specific goals of the qualitative 
component of this research study are present in the following sections. 

How is Hypermedia Used in a "Real-World" Setting? 

Since hypermedia is a new technology, and the approach taken by the CIS hypermedia curriculum 
was novel, researchers wanted to determine and describe in detail how the hypermedia curriculum was used 
in an actual course delivered in a "real-world" setting. 

What Characteristics of Hypermedia Warrant Further Research? 

The purpose of the study was to describe, as much as possible, every aspect of the week-long 
hypermedia training program. Analysis of the data collected would serve as a foundation for identifying 
specific questions and hypotheses about teaching and learning with hypermedia which could be investigated 
in future research. 

Specific goals of the computer tracking tools component of this research study are described in the 
following sections. 

How Frequently and to What Extent Did Trainers and Trainees Utilize Hypermedia 
Materials? 

The CIS Program Operations training course was designed as an integrated hypermedia training 
course. The hypermedia materials were designed to present information, providu guidance and support 
materials for comprehension and higher-level (analysis/synthesis and application activities) learning 
activities. The materials provided resources and tools for both trainer and trainee in the knowledge and 
skills required to function effectively as an executive director of a city-wide or county-wide Cities in 
Schools program. Trainer-facilitated learning activities included role-playing and ^mall-group and large- 
group discussion used in conjunction with and as follow-up to interaction with the hypermedia materials. 
The researchers were interested in the frequency and amount of time specific groups of trainees spent 
actually interacting with the hypermedia materials compared to the amount of time spent in other activities 
away firm the hypermedia workstations. 

Which Paths Did Different Users Employ in Exploring the Hypermedia Materials? 

Structures of hypermedia applications vary widely, from relatively sequential to totally 
nonsequential. The CIS hypermedia application investigated employed a hierarchical structure, in which 
computer, print, or video learning resources were grouped into learning sub-activities which addressed a 
specific learning objective. Ordinarily, trainers directed trainees to examine some or all learning resources 
for a specific learning sub-activity. However, resources for all of the learning activities were available at 
any time, and users could access them by selecting the main screen for any learning activity. In addition, 
user tools in the CIS hypermedia application included a "personal hypermedia list" option, which provided 
the capability to navigate directly to a computer resource which had been added to the user's personal list 
Thus, each trainee's "path" through the hypermedia was potentially different. One purpose of this study 
was to examine the actual trainee use of the hypermedia in a week-long training program. 

What Relationships Were Present Among the Ways Hypermedia Materials Were Used, 
the Extent and Type of Non-Hypermedia Activities, and the Attainment of Specific 
Learning Objectives of the Hypermedia Training Course? 

Specific learning objectives are included for each of the 1 17 learning sub-activities included in the 
CIS Program Operations training course. For each learning objective, matching evaluation strategies are 
presented which specify criteria which would indicate that the learning objective had been met, and where to 
look for such evidence (for example, in trainees' comments during small- or large-group discussions, 
written statements by students made on activity sheets, or the trainees' written summary action plans). 



179 



Learning objectives and learning strategies are clearly specified in the resource book provided to each trainer. 
Since the overall goal of CIS Program Operations training is to help trainees develop the sum of knowledge 
and skills which the learning objectives represent, it was important to demonstrate that this goal had been 
achieved. In addition, the researchers were interested in whether any relationships emerged between the ways 
hypermedia was used in training and the degree to which training objectives wer-, met. For example, would 
a higher proportion of trainees meet learning objectives when trainers made extensive use of the hypermedia 
than when trainers concentrated on other, non-hypermedia training activities? 



Method 

A hybrid methodology was employed in this study. The qualitative methodology employed was 
developed during two pilot teste, then refined for use in the study of a week-long training course. All 
aspects of this use of hypermedia were examined, including the demographics of the participants, trainees' 
and trainers' use of the hypermedia materials, their use of supplemental, trainer-developed materials, and the 
interactions among participants (on-task and off-task, during training sessions, and during social activities). 

Subjects 

Subjects for the study wi e the thirteen trainees and three professional trainers who participated in a 
week-long training course. The trainees, all of whom had volunteered for the course, varied widely in age, 
race, level of education, and amount of experience or knowledge of CIS operations. None of the trainees 
expressed noticeable computer anxiety. 

Trainers for the course were Cities in Schools trainers. All three had extensive professional 
training experience, either with Cities in Schools or with other organizations. All had taught the course at 
least once previously, although their depth of experience with this particular course varied. 

Qualitative Methodologies 

The observer (Nelson) used non-participant observation, a method which is applied by many 
researchers in the social sciences (Delamont, 1992; Miles & Huberman, 1984). The observer took 
extensive hand-written notes while observing training. Observations were recorded in brief descriptions and 
were subsequently expanded upon. The handwritten notes were supplemented by audiotape recording. Tne 
notes and tapes were reviewed each evening, in order to identify patterns and unusual events which should be 
examined more closely on following days. 

The CIS Hypermedia Computer Tracking Tools 

Hypermedia tracking tools were designed into the CIS hypermedia applications, and implemented 
at the same time the application itself was developed. The tools, which were developed as SuperCard 
procedures, continually monitored the status of the hypermedia workstation. Interactions of any kind (key 
press, mouse click, viewing video, etc.) were recorded. Time of day was also continually monitored, and 
the time and duration of all activities was recorded. 

Data from the computer tracking tools was automatically collected in a text file. Information 
collected for each trainee (or pair of trainees working together on the same hypermedia workstation) 
included: 

• the time the computer was first turned on; 

• the start time of each interaction session; 

• the end time of each interaction session; 

complete information on any user input (keypresses, mouse-clicks, etc.), and the time of day of such 
interaction; 

the start time of each period when the computer was not being used; 

the end time of each idle period; 

summary data for each day's hypermedia computer use. . 

Data from the raw text files was transported into a spreadsheet, and cross-tabulations of total usage 
and tables of usage patterns were gent rated for each computer workstation. These data were then analyzed in 
order to identify possible patterns of use. 
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Results 



Results of the Qualitative Component of the Study 

Documentation of training events in the form of extensive logs and other information collected 
during the comse of the five-day training course provided a wealth of data. Analysis of these data indicated a 
number of topics which will warrant closer examination in future studies. These areas for further 
investigation were described in detail elsewhere (Nelson, 1994; Nelson and Harvey, 1994), and will be 
summarized here. 

Diversity of the trainees and grouping issues. The wide variation in background and 
experience suggests that grouping trainees of similar backgrounds and experience levels should be 
investigated. 

Cooperative learning. Trainees working in pairs appeared to exhibit more frequent and higher 
quality interpersonal interactions. Requiring grouping may promote cooperative learning and increase 
training effectiveness. 

Presentation of objectives. Trainers were inconsistent in overtly presenting objectives of 
learning activities. Further study is needed on whether consistently and clearly presenting objectives will 
increase the probability of the attainment of those objectives. 

Formal evaluation of learning. Trainers varied widely in the degree to which they 
systematically applied recommended evaluation strategies. Since many of the evaluation strategies focused 
on comments during group discussions, it wai difficult to assess individual attainment of objectives. 
Methods are needed which will provide evidence of the degree to which each trainee meets training 
objectives, while at the same time not hinder ng the flow of interpersonal interactions and sharing of 
learning. 

Applying learning to real-world situations. Activities designed to help trainees 
systematically apply learning from training ;o their own job situations (by developing a comprehensive 
"action plan") were implemented sporadically by trainers. Further study of trainees after they have returned 
to their own job situation would indicate whether the results of training are having a significant long-term 
effect on trainees' behavior. 

Use of supplemental trainer- developed materials. On several occasions, trainers 
implemented training activities of their own design in lieu of the activities included in the hypermedia 
curriculum. The content and quality of the«:e materials varied widely. It remains to be seen whether the 
Hawthorne effect potential of trainer's usin.g materials they developed themselves will overcome the uneven 
quality of those materials. 

Use of print materials. Although the CIS hypermedia course attempted to integrate the use 
of computer, video, and print materials (e.g., by including a "Print resource" icon on each computer 
resource screen which dictated specific pages of print manuals), trainers only occasionally directed trainees to 
these materials. A study in which the use of print materials is carefully controlled and monitored would 
validate the usefulness of print materials in the context of hypermedia training. 

Variations in trainers' styles. Trainers varied in the degree of apparent comfort with and 
agreement with the principles of hypermedia training incorporated into the training program. A study 
which would pre-assess trainers' preferred instructional styles, monitor the styles actually employed, and 
measure the degree of trainees' attainment of objectives, could identify the trainer styles which were most 
effective in specific situations and, more importantly, those trainers' styles which may impede trainees' 
learning. Such a study would of necessity include a larger number of trainers, and may be difficult to 
implement, since control of variations among trainee groups would be problematic. 

Time management. Trainers in this particulJir course appeared to have difficulties managing 
training time. Estimates of time needed for specific learning activities often proved inaccurate. Logistics 
(e.g., limiting time spent on breaks to the specified period) also impacted the training schedule. In this 
case, it appears that additional trainer training is warranted, rather than a formal investigation, since the 
deleterious effects of poorly managed training time are obvious. 

Course evaluation. Trainers evaluated trainees' perceptions of the hypermedia training course 
by daily "process check" open discussions and by a Li'tert-type reaction form on the final day of training. 
Each of these methods have significant shortcomings. More valid and more reliable course evaluation 
methods, need to be developed and tested in future studies of hypermedia training. 
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Results of the Computer Tracking Component of the Study 



The computer tracking component of this study was more exploratory in nature than the qualitative 
component The computer tracking tools produced a voluminous amount of raw data. Because of the vast 
amount of data, data reduction and subsequent analysis have proceeded at a slower pace than originally 
expected. Preliminary results of those parts of data analysis which have been completed to date are 
presented in the following section. The completed results, when they have been compiled, will be reported 
in a future paper. Data are simply reported here. No conclusions are drawn from them. 

Frequency and extent of use of hypermedia materials. Table 1 presents the amount of time trainees 
at each of six hypermedia workstations spent actually interacting with the hypermedia training materials and 
the total training time. The amount of time trainees spent actually interacting with the hypermedia 
materials was surprisingly low, ranging from no time spent using hypermedia software by one trainee on 
one day to a high of 20%. The overall mea-- usage time for all trainees over the first four days of training 
was 8%. 



Table 1 

Amount of Time Hypermedia Computers Were Used 
as Percentage of Total Training Time 



Trainee No. 


Davl 


Day 2 


Day 3 


Day 4 


Mean 


Std. Dev. 


1 


9% 


18% 


3% 


6% 


9% 


7% 


4 


17% 


10% 


12% 


4% 


11% 


5% 


5 & 10 


12% 


6% 


10% 


4% 


8% 


4% 


6&9 


20% 


9% 


9% 


4% 


11% 


7% 


8 


16% 


2% 


7% 


3% 


7% 


6% 


11 


11% 


0% 


6% 


3% 


5% 


5% 



Note 1. Day five of training was a half-day session devoted to trainees' presentation of action plans 
and course evaluation. Only one trainee used the hypermedia on day 5 for eight minutes. 
Therefore, analysis of day 5 usage was dropped. 

Note 2. Overall mean = 8%; S.D. = 6%. 



Table 2 presents the beginning and ending times of interactions! with the hypermedia engaged in 
by trainees during the first four days of training. This table presents, in tabular form, a picture of the usage 
patterns of trainees. 
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Table 2 

Usage Patterns of Hypermedia Computers Daring Training 



Trainee 


Day. 


Start 1 


End 1 


Start 2 


End 2 


Start 3 


End 3 


Start 4 


End 4 


Start 5 


End 5 


























1 


1 


11:07 


11:41 


12:46 


12:53 


1:05 


1:12 












2 


10:20 


10:57 


4:07 


4:18 


4:51 


4:56 


5:30 


5:43 


5:56 


5:58 




3 


9:50 


9:59 


10:11 


10:12 


11:03 


11:04 












4 


9:40 


9:41 


10:50 


11:11 






































4 


1 


11:05 


11:44 


2:43 


3:09 


4:47 


4:58 


5:09 


5:11 


5:46 


5:56 




2 


10:24 


10:53 


1:33 


1:35 
















3 


11:53 


12:06 


12:12 


12:47 
















4 


10:53 


11:12 


12:41 








































5&10 


1 


10:59 


11:34 


2:34 


2:58 
















2 


10:12 


10:33 




















3 


10:15 


10:22 


10:36 


10:37 


12:01 


12:35 












4 


10:44 


11:03 










































6&9 


1 


11:04 


11:41 


12:18 


12:38 


2:40 


3:06 












2 


10:19 


10:55 




















3 


12:09 


12:45 




















4 


10:50 


11:09 










































8 


1 


11:01 


11:34 


2:34 


3:00 


4:38 


4:54 












2 


1:39 


1:53 




















3 


9:33 


9:34 


12:03 


12:33 
















4 


10:49 


11:04 










































11 


1 


10:58 


10:59 


2:40 


2:54 


4:39 


4:54 


5:39 


6:03 








2 


























12:05 


12:11 


12:29 


12:34 
















4 


10:43 


10:54 


11:15 


11:16 













Relationships among use of hypermedia, use of non-hypermedia activities, and 
attainment of specific learning objectives. Unfortunately, this portion of the study turned out to 
be the least effective of the entire study. The major problem which hindered collection of comprehensive 
and valid data relative to the attainment of learning objectives was that the trainers varied widely in the 
degree to which they attended to the learning objectives specified in the trainer's resource book for each 
learning activity, and the extent to which they carried out systematic application of the suggested evaluation 
strategies. On some occasions, trainers presented the learning objective for a learning activity they were 
facilitating, and reminded trc'nees that the they could remind themselves of the learning objective by 
clicking on the "Objectives" :on which was always present on the computer's screen. Other trainers did 
not present the objective expl citly, and in some cases their behavior indicated that they had not understood 
or were not aware of the learning objective. Applications of the evaluation strategies was sporadic at best. 
For these reasons, it was not possible to continue the analysis of the relationship between uses of 
hypermedia and the attainment of specific learning objectives. 
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Analysis of hypermedia paths used by trainees. Data on the type of hypermedia paths 
used by various trainees are in the process of being analyzed. Preliminary analysis of the data indicates that 
trainees often traveled extremely complex paths while using the hypermedia training materials. 
Categorization of these paths has proven difficult and time-consuming. A number of tools are under 
development to facilitate categorization of hypermedia paths. These tools, and the results of their 
application, will be described in a future paper. 



Discussion 

The volume of information collected through the qualitative component of the study proved 
challenging to summarize and comprehend. The results of the data analysis process, however, met the goals 
of the research study, in that a clear and comprehensive picture of what happened during a five-day 
hypermedia training program was developed. The comprehensive picture provided a foundation for 
identifying a number of important questions which are appropriate for follow-up study. Several of these 
questions can be expressed as null hypotheses which can be investigated by applying quasi-experimental 
techniques. The qualitative research has provided the context and laid the groundwork for more purely 
quantitative research. 

Although the qualitative data had indicated that trainees spent relatively little time actually 
interacting with the hypermedia software, the picture of usage provided by the computer tracking data was 
surprisingly low. An overall mean usage time for all trainees over the first four days of training of 8% 
could indicate that trainers were not encouraging trainees to make appropriate use of the hypermedia. 
Qualitative data indicated that, in several instances, trainers presented content identical to that included in the 
hypermedia using overheads, flip charts, and lecture. On the other hand, it should be kept in mind that the 
hypermedia training materials studied were designed as integrated hypermedia to be used to support 
interpersonal interactions (role playing, small and large group discussions, etc.) which clarified and extended 
the information presented in the hypermedia. Follow-up activities of this type, whether appropriate or 
inappropriate, would not show up in the tracking data as time spent using the hypermedia, even though the 
activities clearly were extensions of the learning begun through interaction with the hypermedia materials. 

Analysis of data generated by the computer tracking component of the study proved more 
challenging. The body of information is extremely large and extremely complex, particularly in regards to 
the depth and variety of hypermedia paths traversed by users. The flexibility provided by hypermedia 
materials clearly has both positive and negative aspects. In order to identify these aspects, and to optimize 
the use of hypermedia materials, it is necessary first to develop a taxonomy of types of paths which are used 
by different trainees. The relationships among types of paths, learning styles, and learning outcomes can 
then be investigated more systematically. One promising approach to developing such a taxonomy would 
be to examine the techniques developed by social scientists to categorize and analyze interactions in small 
groups. 

The inability to formally examine the relationships among uses of hypermedia, uses of other non- 
hypermedia training activities, and the attainment of specified learning objectives was disappointing to the 
researchers. The researchers were also key members of the team which had designed and developed the 
hypermedia training course, and had worked diligendy to develop meaningful objectives and appropriate 
evaluation strategies for these objectives. In addition, the researchers were keenly aware of the importance 
placed on achieving specific learning objectives by the Cities in Schools administrators who coordinated the 
OS hypermedia development project. Cities in Schools had invested considerable resources in the 
hypermedia training curriculum, and it was important to demonstrate that these resources had been invested 
wisely. 



Conclusions 

This qualitative study of a hypermedia training program treated a particular group of trainees and 
trainers engaged in a particular training program as a "culture" to be investigated tnrough intensive and 
careful observation. The data gathered, as in most qualitative studies, is voluminous, and requires extensive 
and careful analysis in order to extract useful patterns and to draw meaningful conclusions. Since each 
group of trainees and trainers and each training program application is different, it is not appropriate to 
extrapolate the results of this study to all training programs using hypermedia application. 
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Qualitative studies are, however, extremely useful for providing a comprehensive picture of what 
actually happens in a "real-world" training situation. This picture provides a solid foundation on which to 
develop questions for further study, which can then be investigated sj jtematically using quasi-experimental 
methodologies. 

Computer tracking tools, to be implemented, require considerable programming expertise and 
effort. In addition, they are most efficiently implemented if they are incorporated into the design of the 
hypermedia application and implemented as the hypermedia application itself is developed. Difficulties 
inherent in allocating scarce project resources to develop hacking tools, including the challenge of 
incorporating tracking tools while meeting pressing delivery due dates, makes it difficult to implement 
tracking tools on a wide basis. In addition, it is extremely difficult to add tracking tools on a post hoc 
basis, especially by people other than the group who originally designed and developed the hypermedia 
application. For these reasons, tracking tools have not, to date, been used extensively in hypermedia, 
research. 

These researchers believe that computer tracking tools hold great promise for providing insights 
into how hypermedia applications are actually used and how they can be used more effectively. These tools 
are particularly effective when their use is combined with other research methodologies, such as qualitative 
investigations of specific hypermedia training courses. Both qualitative research methodologies and 
computer tracking methodologies hold great promise for furthering understanding of how hypermedia can be 
designed for effective training. Both have the advantage of being applied in "real-world" settings, studying 
actual applications of hypermedia as they are actually being used without distorting that application by their 
presence. When used in combination, qualitative research methodologies and computer tracking 
methodologies provide a synergistic extension of the information provided by each, and therefore provide a 
comprehensive and powerful foundation on which to base future decisions about the design and delivery of 
hypermedia-based training. 
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